New Filebeat subcommand: generate #9314

kvch · 2018-11-30T16:24:08Z

This PR exposes the existing Filebeat module generators as generate subcommand.

$ ./filebeat generate -h
Generate Filebeat modules, filesets and fields.yml

Usage:
  filebeat generate [command]

Available Commands:
  fields      Generates a new fields.yml file for fileset
  fileset     Generates a new fileset
  module      Generates a new module

The subcommands module, fileset and fields use the same code as the scripts.

I added E2E tests to validate the functionality.

Blocked by #9147 as it contains the refactorings of that PR.

ph · 2018-12-03T13:27:14Z

filebeat/cmd/generate.go

+ }
+
+ genModuleCmd.Flags().String("modules-path", defaultHomePath, "Path to modules directory")
+ genModuleCmd.Flags().String("es-beats", defaultHomePath, "Path to Elastic Beats")


Do we really need to expose both, because by default both of the flags will point to the same place?

Hmm... Not yet. Btu when community Beats will support a similar module-handling, we will need this.

kvch · 2018-12-03T13:52:50Z

jenkins test this

ph

LGTM, WFG

kvch · 2019-01-08T10:35:57Z

Rebased

ruflin · 2019-01-09T10:17:59Z

If we expose this to the user, I think we should also add some docs on how we expect the users to use this. How will the user handle the new fields?

The PR has needs_backport label. Is there a reason we need this in 6.x?

kvch · 2019-01-09T10:28:55Z

👍 on the docs, that's why the PR has the needs_docs label. What do you mean first question?

Obviously, we don't "need" it, but we planned to release it in 6.x. Why do you think it is problem?

ruflin · 2019-01-09T10:35:54Z

A user creates a module with this command, how will he load / configure the additional fields that are used for it? If the user upgrades to a newer version, will the module stay intact? What if he wants the module on multiple machines?

kvch · 2019-01-16T17:54:33Z

I am still not sure if I follow you. The behaviour of the generators has not changed. In this PR I only added new commands and subcommands to Filebeat to make existing functionality accessible to more people.

After a user generates a module and a fileset, he/she can add the fields of the fileset to the appropriate fields.yml file. generate fields is not able to produce a fields.yml which is 100% guaranteed to match the actual fields. It is just a "crutch" for a user to get started with the file. The generator explicitly asks the user to check the result. It also generates fields with the description which need to be filled in.

Why wouldn't the module stay intact during an upgrade? If the name of the user's module is the same as a new module coming with the new release, it gets overwritten. But this has been the behaviour since introducing module generators in Filebeat.

If he/she wants to have this module on multiple hosts, it is possible to copy them as before. Nothing changes here, also.

andrewkroh · 2019-01-16T19:26:34Z

Nice addition to Filebeat. An accompanying filebeat test module command that uses the _ingest/pipeline/_simulate API to test the pipeline and validate the resulting document would also help make module development more accessible.

ruflin · 2019-01-17T12:32:34Z

So far this functionality was only accessible to devs in the Beats repo and was not really intended for production but more for module development. With the above we make it accessible to all users and production use cases.

One question I don't fully understand yet is what happens with the fields.yml. If the above commands are used in a dev environment and make update is run, they are put into the global fields.yml file and fields.go inside the binary. If someone uses the above command outside the dev environment, he gets the fields.yml in the module (which we don't ship) but AFAIK it's not really usable and will not have any effect on the template.

To be clear: I really like to make the above commands available to all our users but I want to make sure they have the experience they expect when they run the commands.

kvch · 2019-01-18T08:55:04Z

Some parts of make update are not yet ported to Golang. So it is not yet possible to generate everything for a module.
@ruflin I am OK with putting this PR on hold until all generators can be exposed. WDYT?

ruflin · 2019-01-18T09:02:02Z

@kvch Even if mage update would be all in Go, there would be still an issue here. The reason is that by default even if there is a fields.yml, the fields from the binary will taken. Even if a user adds his fields to the global fields.yml, it will not read by default.

To keep this moving forward few ideas:

We recommend users to copy their fields to append_fields
We tell users that fields.yml does not work yet
We introduce some logic in the module that can read the local fields.yml in the module and append it to the template if it is there.

To get this PR in quickly, we could start with 2 and see what the user feedback will be and work on 3. I personally like 3 because it will make modules "just work".

kvch · 2019-01-18T09:48:13Z

@ruflin Indeed. I like what you have proposed in option 3. I will open a new PR with the changes which lets users read their local fields.yml file. That is a nice addition on its own.

kvch · 2019-01-18T09:48:46Z

I only need to investigate why the test cannot find the test pipeline on Windows.... >.<

kvch · 2019-02-01T15:42:18Z

jenkins test this

kvch · 2019-02-04T18:48:46Z

jenkins test this

kvch · 2019-02-08T14:58:24Z

Failing tests are unrelated.

kvch · 2019-02-08T14:59:33Z

Needs backport to 7.x branch...

kvch added review Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. blocked needs_docs labels Nov 30, 2018

kvch requested a review from ph November 30, 2018 16:24

kvch force-pushed the feature-filebeat-expose-generator-cmd branch from 016f4c2 to a220213 Compare December 3, 2018 08:30

ph reviewed Dec 3, 2018

View reviewed changes

ph approved these changes Dec 3, 2018

View reviewed changes

kvch removed the blocked label Dec 3, 2018

kvch force-pushed the feature-filebeat-expose-generator-cmd branch from 40f7599 to e7901be Compare January 8, 2019 10:35

kvch requested a review from a team as a code owner January 8, 2019 10:35

kvch removed the request for review from a team January 8, 2019 10:46

kvch force-pushed the feature-filebeat-expose-generator-cmd branch from fa4afbc to ea07b1a Compare January 9, 2019 09:07

kvch removed the needs_backport PR is waiting to be backported to other branches. label Jan 15, 2019

kvch mentioned this pull request Jan 16, 2019

New subcommand: generate #6912

Closed

2 tasks

kvch force-pushed the feature-filebeat-expose-generator-cmd branch from d4baa47 to 649e95a Compare February 4, 2019 08:41

kvch added 5 commits February 4, 2019 15:54

add new subcommand generate to filebeat

1ee880a

fix paths in tests

6d117cc

fix more path

19f05e4

fix fmt

db5410a

tmp

a30035d

kvch force-pushed the feature-filebeat-expose-generator-cmd branch from 649e95a to a30035d Compare February 4, 2019 14:55

kvch merged commit cc9a326 into elastic:master Feb 8, 2019

kvch added the needs_backport PR is waiting to be backported to other branches. label Feb 8, 2019

kvch mentioned this pull request Feb 11, 2019

Support external fields.yml files #10676

Closed

kvch removed the needs_backport PR is waiting to be backported to other branches. label Aug 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Filebeat subcommand: generate #9314

New Filebeat subcommand: generate #9314

kvch commented Nov 30, 2018

ph Dec 3, 2018

kvch Dec 3, 2018

kvch commented Dec 3, 2018

ph left a comment

kvch commented Jan 8, 2019

ruflin commented Jan 9, 2019

kvch commented Jan 9, 2019

ruflin commented Jan 9, 2019

kvch commented Jan 16, 2019

andrewkroh commented Jan 16, 2019

ruflin commented Jan 17, 2019

kvch commented Jan 18, 2019

ruflin commented Jan 18, 2019

kvch commented Jan 18, 2019

kvch commented Jan 18, 2019

kvch commented Feb 1, 2019

kvch commented Feb 4, 2019

kvch commented Feb 8, 2019

kvch commented Feb 8, 2019

New Filebeat subcommand: generate #9314

New Filebeat subcommand: generate #9314

Conversation

kvch commented Nov 30, 2018

ph Dec 3, 2018

Choose a reason for hiding this comment

kvch Dec 3, 2018

Choose a reason for hiding this comment

kvch commented Dec 3, 2018

ph left a comment

Choose a reason for hiding this comment

kvch commented Jan 8, 2019

ruflin commented Jan 9, 2019

kvch commented Jan 9, 2019

ruflin commented Jan 9, 2019

kvch commented Jan 16, 2019

andrewkroh commented Jan 16, 2019

ruflin commented Jan 17, 2019

kvch commented Jan 18, 2019

ruflin commented Jan 18, 2019

kvch commented Jan 18, 2019

kvch commented Jan 18, 2019

kvch commented Feb 1, 2019

kvch commented Feb 4, 2019

kvch commented Feb 8, 2019

kvch commented Feb 8, 2019